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Cost Analysis: A Starter Kit 


Why was this starter kit developed and who is it for? 


IES encourages, and sometimes requires, cost analysis for IES-funded research and 
evaluation projects. IES believes cost analysis is crucial to support education agencies 
decision-making in the adoption of programs, policies, or practices. 


? 


IES has released this starter kit for grant applicants who are new to this kind of 
analysis. The kit will help you plan a cost analysis, setting the foundation for more 
complex economic analyses. We outline a three-phased approach, which we have 
broken into additional steps. 


Do you have feedback on this document? This is the first version of the IES Cost 
Analysis Toolkit. IES will update it regularly as readers find errors and offer 
improvements. We would appreciate your feedback on this resource. Comments can 
be sent to ncee.feedback@ed.gov. 
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Quick Start Checklist 


This quick-start checklist summarizes the major steps in a cost analysis. You may 

not be familiar with each of the terms it uses until you have read this Toolkit. We 
encourage you to read the Toolkit in its entirety and then return to this checklist when 
you are ready to begin your first cost analysis. 


Step 1: Clarify the program model 


O The program has a clear theory of change. 


O A logic model depicts the theory of change, clearly identifying how core 
components of the program work together to influence outcomes. 


Step 2: Choose the perspective(s) 


O Ihave checked which perspective(s), if any, is required or encouraged by the 
IES grant program to which I am applying. 


O Tam clear about the intended audience(s) for my research and its cost analysis 
and have considered audience needs in determining which perspective(s) will 
be useful. 


O1 Ihave chosen a perspective that best fits the purpose of my research while also 
being feasible for me carry out. If I have not undertaken an analysis from the 
societal perspective, I am clear about the relative advantages and disadvantages 
of that approach. 


1 The perspective I have chosen is clear in my description of the analysis. 


Step 3: Identify the quantity and quality of the program’s key ingredients 


O I used different data collection approaches as needed to determine what 
resources were used by the program, including those that might not have been 
originally anticipated by its developer. 


O1 Ihave considered each main category of typical ingredients and identified the 
specific ingredients used by the program. 


O Ihave recorded program ingredients in enough detail, in terms of quantities 
and qualities, so that others will understand what it takes to implement the 
program. 
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O Iconsidered any important induced behaviors of the program and added 
relevant resources to my list of ingredients. 


Step 4: Identify or estimate a price for each ingredient 
O Iam prepared to conduct parallel cost analyses using both national and local 
price data. 


O1 If there is a constraint on my choice of prices, I have determined whether 
national or local prices make the most sense given the specific purpose and 
target audience of my cost analysis. 


O Ihave assigned prices to each ingredient in my analysis using CostOut or other 
sources. 


O Ihave ensured that units of both price and quantity align. 


Step 5: Adjust costs for your context 


C1 Ihave made contextual adjustments, such as geographic or present value 
adjustments, relevant to the program I am examining. 


Step 6: Calculate costs 


O Ihave presented costs in a way that fits the needs of my audience. 


O If have a multiyear program, I have considered whether the costs change year 
to year and presented the results accordingly. 


Step 7: Conduct sensitivity analysis 


O1 Ihave considered the biggest cost drivers of my program and whether they 
should be included in a sensitivity analysis. 


Step 8: Determine whether to use the results for further economic 
evaluation 


O Iam aware of both strengths and weaknesses of the economic evaluation 
methods I am using. 


O [have referenced the current IES Requests for Applications and I know if there 
are requirements for a specific type of economic evaluation for my grant. 
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Introduction 


What is cost analysis? 


Cost analysis examines the costs of specific programs—including interventions, 
education practices and policies, and tools—designed to improve students’ academic 
outcomes. Cost analysis— 


e Describes the resources needed to run a program, whether related to 
personnel, facilities, or materials 


e Attaches value (estimates prices) for those resources 


¢ Gathers information to estimate a program’s cost 


What is the purpose of cost analysis? 


Researchers analyze cost to help state and local education providers decide how 
to best allocate their resources. This could include adopting new programs, 
implementing new instructional strategies, or deciding to continue an existing 
curricular approach. 


Cost analysis can also— 


¢ Identify the resources for a specific program. Cost analysis can help schools 
and districts understand how to implement a program with fidelity. 


e Surface issues of implementation and accountability. Cost analysis can 
inform whether, under current policies and practices, resources are used as 
planned. 


Cost analysis is often a building block for further economic evaluation, including cost- 
feasibility analysis (CFA), cost-effectiveness analysis (CEA) and cost-benefit analysis 
(CBA). These other approaches can— 


e Clarify whether an education agency has the resources available to 
implement a program. By comparing the resources needed to implement a 
new program and the costs to procure them with the existing resources and 
available budget, decision makers can determine whether they prefer to adopt 
a new program or continue or scale up an existing one. 
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Inform choices among different programs or strategies. Choosing the option 
that does the most good per dollar spent requires cost information, as well as 
results from the specific programs or strategies. 


Support making decisions about existing programs. When cost information 
about a program is compared to its results or benefits, it can inform whether 
that program should be scaled up or cut back. 


You can find more information about more types of economic analysis later in this 
toolkit, in Step 8. 


What concerns about cost analysis does the starter kit help to tackle? 


Cost analysis is complex and tedious. A basic cost analysis does not need to 
be overly difficult or time-consuming. You should identify and price the most 
important ingredients, whether staffing, facilities, or materials. Less costly 
inputs can often be approximated, since their accuracy is less important. A 
good cost analysis is about being as accurate as you can be—being “within the 
ballpark”—with the analytic resources you have. The time-saving tips included 
in this starter kit can help you do that. 


Cost analysis is an additional, distinct process from other forms of program 
evaluation. One of the main steps in cost analysis—identifying the required 
resources—is a critical component of implementation research within a 
program evaluation. By acknowledging this overlap during the study-design 
phase, you can intentionally incorporate cost analysis from the very beginning 
of the research effort, saving time in the long run. 


Cost analysis and the Ingredients Method 


Within the costing field, the term “ingredients” refers to the resources needed to 
implement a program. The approach to cost analysis that focuses on identifying and 
then valuing those ingredients is known as the “Ingredients Method” (Levin, McEwan, 
Belfield, Bowden, & Shand, 2018). 


Are there other methods for doing cost analysis besides the Ingredients Method? 
Practically speaking, no: Any credible cost analysis needs to carefully consider the 
resources—that is, the ingredients—that a program uses. There are ways to help speed 
up that process, however, as we will discuss. 
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Total costs versus incremental costs 


As its name implies, the total cost of an intervention is the sum of the prices of all 
ingredients needed to deliver it. In contrast, the incremental cost of an intervention 
represents only those costs above and beyond one or more alternative conditions. 


For example, if a school spends an hour per day on math instruction and a new 
program is a direct substitute—that is, it requires the same teachers to offer the same 
60 minutes of math daily—there would be no incremental cost associated with teacher 
labor. (There may, of course be other incremental costs. Consider, for example, the 
cost of teacher time spent in professional development to offer the new program.) 


In contrast, if the new program requires an additional ten minutes per day of 
instruction, the additional resources used in support of those ten minutes and the 
prices of those resources would represent incremental costs. 


Finally, if there is no existing program serving the need of the intervention being 
considered, be sure to include all ingredients required to deliver it. In this instance, 
the total and incremental costs are effectively the same. 


Is my program cost-free? 


Some programs appear not to need a cost analysis because they involve no new 
expenditures to implement. For example, a program may be using already-available 
classroom space, involve staff that are already employed, and use computers already 
on hand. Is such a program cost free? 


From a financing viewpoint, given those details, new expenditures may be zero. 


However, from a cost analysis viewpoint, resources are being reallocated to a new 
purpose. These items will not incur any new budget outlays but still have a value 
that you should account for. Therefore, you should still note these resources as a 
cost—even though they do not result in expenditures. Anyone considering adopting 
the program will benefit from full transparency about the resources required to 
implement it properly. This includes being able to ask themselves, “Do we have the 
ability to recruit those ‘free’ resources in our own context?” 


Remember: while decision-makers may care most about new burdens to the budget, 
the purpose of a cost analysis is to document the resources required to implement a 
program regardless of how they are financed. 
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How is the starter kit organized? 


The starter kit is organized into eight steps that represent a basic cost analysis, 
grouped into three phases. Those phases relate to ingredients, prices, and costs: 


¢ Phase 1: Identify the ingredients. This step involves determining what 
resources are used by the program—whether related to personnel, 
facilities, equipment, supplies or other inputs—and then describing them in 
characteristics and quantity. 


e Phase 2: Price the ingredients. Value each program resource by estimating or 
determining their prices for the period. 


e Phase 3: Create and use the cost estimate. Calculate the cost of the program, 
test assumptions (sensitivity analysis), and then decide whether to use the 
results for any further economic evaluation. 


The steps in the table below correspond to the sections of this starter kit. 


Table 1. Steps in a Cost Analysis 


Phase 1: Identify Program Ingredients 


Step 1: Clarify the program model 
Step 2: Choose the perspective(s) 
Step 3: Describe key ingredients 


Phase 2: Price the ingredients 
Step 4: Identify or estimate a price for each ingredient 


Step 5: Adjust costs for your context 


Phase 3: Create and use the cost estimate 
Step 6: Calculate costs 
Step 7: Conduct sensitivity analysis 


Step 8: Determine whether to use the results for further economic evaluation 


Appendices at the end of the starter kit provide additional information and resources. 


e Appendix A includes a blank logic model template. 
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e Appendix B lists additional resources that may be useful as you conduct your 
first cost analysis. 


e Appendix C includes a list of resources from which key content in this starter 
kit was sourced. 


e Appendix D includes a list of the individuals who were interviewed during the 
development of the starter kit. 


A brief example 


Table 2 shows data from hypothetical cost analysis of a one-year program or the first 
year of a multi-year program. We are assuming the program is an add-on to existing 
programs. The program requires additional teacher and classroom time above what 
was already being used within the school. In the first three columns, Phase 1, we 

see those ingredients. The next column, Phase 2, adds prices for each ingredient. 
And finally, Phase 3 multiplies quantity and price to produce a cost per ingredient. 
If we summed up those costs (not shown), we would arrive at a cost estimate for the 
program in year one. 


Table 2. Building Blocks of a Cost Analysis: An Example 


Cost 


Ingredient Quantity Description Price ; ' 
(Quantity x Price) 


Bachelor’s 


120 hours ; 
: . degree with $7,200 
Teacher time per academic $60 / hour 


5 years of (120*60) 


year; 1 teacher ; 
experience 


120 hours 
Classroom per academic $600 
900 sq. ft. $5 / hour 
space year; 1 (120*5) 
classroom 


As you move through the starter kit, each step will provide more information about 
these three phases, exploring some of the nuances of identifying ingredients, pricing 
them, and creating and using a cost estimate. 
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How does the starter kit align to IES grant requirements or 
recommendations? 


IES recommends applicants use this starter kit in conjunction with the specific 
requirements or recommendations of the IES grant they are applying for. 


Cost analysis attempts to estimate the costs of a specific program, not the cost of the 
research. Also, keep in mind that IES is asking you to include a plan for carrying out 
your cost analysis in your application. At a minimum, that plan should include your 
choice of perspective (Step 2) and prices (Step 4), as well as other details related to the 
steps you plan to take to complete the required economic evaluation (Step 8). You are 
not expected to a include the actual analysis of cost in your application. 


What other resources on cost analysis are out there? 


Depending on your needs, you may want to deepen your knowledge about cost 
analysis beyond what this starter kit can provide. Appendix B contains a list of all 
resources referenced in this guide. 


One resource to mention here, however, since it provides a useful big-picture 
overview, is the set of free online videos of an IES-funded workshop on cost analysis 
organized by the Center for Benefit-Cost Studies of Education (CBCSE) at Teachers 
College, Columbia University. They include— 


e The Ingredients Method Part 1 and Part 2 by A. Brooks Bowden 
¢ Pricing Ingredients by Robert Shand 
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Phase 1: Identify program ingredients 


Table 3. Building Blocks of a Cost Analysis: Phase 1 


Ingredient Quantity Description 


Cost 
(OE Tiana. au (a>) 
Quantity in Descriptive 


Ingredient 1 . ; Quantity x Price 
units details 


; Quantity in Descriptive ; : 
Ingredient 2 ; : Quantity x Price 
units details 


Overview 


The first phase of cost analysis focuses on identifying the ingredients (or resources) 
needed to implement a program. Phase 1 asks: What resources are required to 
implement a program with fidelity? The list of ingredients includes their quantity 
and other important characteristics. For simple programs, this phase might be 
straightforward. For others, it will require some work to figure out the key resources 
used by the program. Either way, as we look ahead, this sets the stage for Phase 2, 
which is about attaching prices to those ingredients. 


What steps are involved? 


1. Clarify the program model 
2. Choose the perspective(s) 
3. Identify the quantity and quality of the program’s key ingredients 
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Step 1: Clarify the program model 


Why is this important? 


Logic models depicting a program’s theory of change help identify ingredients 
for a cost analysis, including inputs, activities, and, in some cases, outputs. Logic 
models represent plans, while cost analysis should be based on actual program 
implementation. The outcomes of your logic model can help you plan further 
economic evaluations. 


Developing a logic model to visualize a program’s theory of change 
A logic model describes what a program does with— 


¢ Inputs: The program ingredients (resources), such as funding, staff, or facilities 
e Activities: How the program accomplishes its goals 

¢ Outputs: What the program produces 

¢ Outcomes: Short-term, medium-term and long-term goals 


Figure 1. Key Components of a Logic Model 


te >> sats > oem > omnes > 


What a program aims 
What a program does ; 
to accomplish 


Logic models and costs 


It is worth considering: Does a logic model capture all the resources used by a 
program and therefore all its costs? 


Not necessarily. There may be costs that are overlooked when focusing on a program’s 
theory of change, including unintended costs that were not predicted by the model or 
described by its developer but that are observed in implementation. It is often helpful 
to talk with individuals directly involved in delivering the program to identify these 
otherwise hidden costs. Clarifying the program model is only Step 1 of a cost analysis. 
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Time-saving tip: Timely information collection 


An important time-saving tip: If possible, do not wait until after the program 
ends to gather information about cost and implementation. The process 

will be much more tedious (including tracking down people who recall key 
details), not to mention less accurate. 


This time saver applies not only to gathering information about the program 
model, which is the focus of this step, but also to any information gathering 
needed for the next steps, such as data on program ingredients. Gather 
those data while the program is running. 


An example 


The running example used in this starter kit is the fictional Summing Up program. 
We have adapted details of this fictional program from materials developed by Fiona 
Hollands and Yilin Pan of Teachers College, Columbia University and presented by 
Robert Shand of American University at an IES-funded workshop on cost analysis. 


Summing Up is a computer-assisted math program where students use adaptive 
software to practice their math skills. The setting for its evaluation was a public school 
within the United States in academic year 2018 to 2019. It involved 100 students in the 
third grade. Outcomes were measured using the STAR math assessment. As part of the 
intervention— 


e Five groups of 20 students practiced their math skills using Summing Up for 30 
minutes per school day 


¢ One of five teachers supervised each group to check students’ progress and 
answer questions 


e The teachers participated in a four-hour training session 
e The program license fee was $100 per student and included the training fee 


¢ Purchasing the software required 5 hours of district administrator time 
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Logic model using Summing Up 


The questions that motivate a cost analysis of Summing Up might be: What was the total cost of implementing the 
program and the average cost per student? We begin examining that question by clarifying the program model through 
the creation of a logic model. Table 4 shows an example what a logic model might look like for Summing Up. 


Table 4. A Logic Model for Summing Up 


Summing Up 


adaptive software 


Five teachers’ 
time (for program 
implementation 
and training) 
Program license 
fees 
Administrator 
time (to buy 
software) 
Classroom/lab 
space 
Computer for 
each student 
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Activities 


Students use 
software to 
practice their 
math skills (30 
min. per day) 
Teachers take 
part in training 
Teachers 
supervise student 
use of software 


Outputs 


100 students 
engage in 90 
hours of adaptive 
math instruction 
per year 

5 Teachers 

are trained in 
implementing 
Summing Up 


Outcomes 


Short term 


Students 

have better 
understanding 

of the math 
concepts covered 
in the curriculum 


Long term 


Students have 
higher STAR 
math assessment 
scores at year- 
end 
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Key points 


e Start a cost analysis by clarifying a program’s purpose and theory of change. 
¢ Create a logic model, a visual representation of a program’s theory of change. 


e Clarify the program model with program documentation, interviews with 
experts or implementers, previous studies, and direct observations via site 
visits. 


¢ Document a program’s theory of change for strong grant applications. 


Checklist 


O The program has a clear theory of change. 


O A logic model depicts the theory of change, clearly identifying how core 
components of the program work together to influence outcomes. 


Additional resources 


A blank logic model template is provided in Appendix A that you can print and use. 
The IES Regional Educational Laboratories (RELs) provide additional examples of 
logic models here and here. 
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Step 2: Choose the perspective(s) 


Why is this important? 
This step is about deciding: Whose costs are we measuring? 


While all cost analyses focus on measuring how much a program costs, there are 
different “lenses” or perspectives that one could adopt in doing that analysis. 
Perspective determines which ingredients will require you to assign prices to them. 
While you may not assign prices to all ingredients, it is still important to document in 
your write-up every ingredient used in a program to aid future replication. 


What are the main perspectives? 
Three common perspectives of cost analyses are— 


¢ Societal perspective: From this perspective, all resources used by a program 
have value and are counted—regardless of who pays for them and even if they 
are provided in kind. Every resource needs to be assigned a price. That includes 
resources where the “price” is the opportunity cost, meaning the forgone gains 
of using the resource for another purpose—a concept we will discuss in Step 3. 


e School perspective: Count only the school resources used to implement the 
program. Time spent by parents or program volunteers would not be counted, 
since the school does not pay for their time. 


e District perspective: Any district resources used to implement a program 
would be included. This includes, for example, the time spent by a central 
office program director to monitor implementation across schools. 


For higher-education studies, another perspective could be— 


e Student perspective: How much does it cost the student to participate in 
a program? That includes not only direct costs, such as fees, but also the 
opportunity cost of their time. Note that cost analyses of K-12 programs in 
developed countries generally do not include a student perspective, since it is 
less relevant in that context—students at that age are usually not able to choose 
between school and work. 
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How costs vary by perspective 


Choosing the perspective(s) of a cost analysis typically happens at the start of the 
process. For illustrative purposes, we will assume you have already calculated the 
value in Year One of each of the additional resources above and beyond business as 
usual used by a certain program. The incremental costs for a program for struggling 
readers are shown in the table below 


Table 5. Incremental Costs of a Fictitious Reading Program 


: Cost (per 
Ingredient Payer 
student) 


Training time for teachers $250 
Fees for teacher training 


Volunteer time using minimum wage to estimate opportunity 
; $170 Volunteers 
cos 


So, how much does this program cost? It all depends on the perspective. 


¢ From a societal perspective, all of the costs would need to be included. Adding 
them together, we arrive at a program cost of $495 per student. 


¢ From a school perspective, the program cost would be only $325. This is 
because the value of the volunteer time would not be included, since it is free 
to the school. 


As this example shows, the choice of perspective can have an important effect on the 
answer to the question, “How much does this program cost?” 


Choosing the perspective(s) 


Ideally, anyone doing a cost analysis would calculate and convey multiple 
perspectives, starting with the societal one. Multiple perspectives make your findings 
relevant to the broadest range of decision makers. Including a societal perspective 
allows for the most comparability between your study and those of other researchers. 
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Other perspectives may be needed to be responsive to different key stakeholders. 


For example, in studying a district-based reform, you should include both societal 
and district perspectives. That way decision makers at different levels would have cost 
information that is the most relevant to them. It would also make your analysis the 
most comparable to existing studies of that type of reform, some of which might use 
one perspective or the other. 


When you only have time or resources to take one perspective, however, the key 
question should be: How will my analysis or study be used, and by whom? 


For instance, if you are helping principals choose among common options that only 
rely on school-based resources, then the school perspective will be the most relevant. 
If you are informing a broader set of decision makers, including research that could 
be used nationally, then a societal perspective will convey all the program costs. Of 
course, once you have calculated costs from a societal perspective, identifying costs 
from more narrow perspectives is relatively easy, since you would simply not count 
costs considered in that perspective. 


Time-saving tip: Your choices of perspective(s) 


If you are new to cost analysis or have limited resources to undertake the 
analysis, a school or district perspective could be a useful approach, since it 
is the simplest to undertake. However, you need to consider the downsides 
involved. 


For example, not including a societal perspective may make your analysis 
less relevant to a broader set of decision makers. Moreover, different 
education agencies split costs between schools and districts differently, so 
choosing a school perspective might not produce generalizable information. 


Even if you choose to take a school or district perspective, you still should 
identify all of the ingredients used in the intervention (but not their 

costs if they are paid from other sources) so that others will have a fuller 
understanding of the intervention’s ingredients and will be able to calculate 
the full costs. 
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Let us continue the example of Summing Up, the fictional computer-assisted math 
program we introduced in Step 1. What perspective or perspectives might a researcher 
choose who is studying Summing Up? If the researcher were, say, employed by the 
school or the district to evaluate their programs, such as Summing Up, they might 
choose to take a narrower perspective of the school or the district. 


More likely, however—especially if this research were part of an IES research grant— 
the researcher is aiming to generate insights that will be used more broadly, such 

as in their region or even nationally. In that case, they will want to use a societal 
perspective for their cost analysis. They might choose to also include a school 
perspective to ensure that their findings are also relevant to principals and other local 
decision makers that might use the research. 


IES research grants and the choice of perspective(s) 


For IES grant applicants or recipients, a key factor in choosing the perspective(s) of a 
cost analysis is being responsive to any required or recommended approaches for that 
grant. A good place to start is by examining the grant RFA and supporting documents 
for any guidance or requirements related to cost analyses. 


IES recommends that researchers use the societal perspective. Student, school, and 
district perspectives can be derived from the societal perspective, giving applicants 
maximum flexibility in conducting economic analysis and consumer maximum 
flexibility in making use of it. 


Key points 


¢ Choosing a perspective, or set of perspectives, for your cost analysis is about 
deciding whose costs you will focus on. 


¢ A societal perspective is the broadest perspective and, therefore, has the 
advantage of being transparent about all costs associated with a program. 
A school (or district) perspective, which can be derived from a cost analysis 
initially developed at the societal perspective, provides information that is 
directly relevant to decision makers at that level. 


e JES recommends analysts initially take the societal perspective. Your choice 
of perspective may vary depending on the purpose of your research and your 
target audience of decision makers. 
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Checklist 


O Ihave checked which perspective(s), if any, is required or encouraged by the 
IES grant program to which I am applying. 


O Iam clear about the intended audience(s) for my research and its cost analysis 
and have considered audience needs in determining which perspective(s) will 
be useful. 


O1 Ihave chosen a perspective that best fits the purpose of my research while also 
being feasible for me carry out. If I have not undertaken an analysis from the 
societal perspective, I am clear about the relative advantages and disadvantages 
of that approach. 


O1 The perspective I have chosen is clear in my description of the analysis. 
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Step 3: Describe, in quantity and quality, the 
program’s key ingredients 


Why is this important? 


This step is about describing what resources are needed to implement and carry out 
the program you are studying. This is at the heart of a cost analysis, since identifying 
the ingredients (this step) and pricing those ingredients (the next step) are the basic 
building blocks of a cost estimate. 


Time-saving tip: Consider implementation surveys 


Time saving tips: Save time by including questions identifying program 
ingredients and their costs into implementation surveys—that is, any 
already-planned initial data collection. By weaving those questions in, you 
could obtain information for Step 3 with little additional effort. Seeing your 
cost analysis as part of your program evaluation, and not an add-on, is 
fundamental. 


Sources of information on ingredients 


Where can you find information on a program’s ingredients? It can come from many 
different sources: 


¢ Reports and descriptions of the program, including information from program 
developers or previous evaluations 


e Interviews you conduct with those involved in developing, implementing, 
monitoring or evaluating the program, such as program directors, school 
personnel, parents and (for program serving adults) program participants 


e Survey data you collect from individuals involved in program implementation 
e Your direct observation of the program 
¢ Program budget information, which we discuss in more detail below 


Whatever the source, the focus should be on determining the ingredients actually 
used by the program, rather than what was planned. Moreover, for programs with 
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multiple sites, you will need to identify a sample of sites from which to collect 
information on ingredients—ideally the same sites where effectiveness data are being 
collected for an impact evaluation. This may result in a range of resource use and 
associated costs across sites. In addition to presenting averages, it may also be useful 
to present a low and high estimate of resource use and costs across sites. 


If there is a lack of consensus about the key ingredients of a program-—say, different 
sites used different key ingredients—you can explain that in your analysis and present 
results from cost estimates using different assumptions about the key ingredients. 


For some impact evaluations, business as usual for the control group may be 
comprised of a variety of practices. In this situation, you would ideally obtain 
information on ingredients used in each of these control group programs. This may 
not be feasible given your available resources; specific suggestions for addressing this 
concern are discussed more below in Step 4. 


Using program budgets to identify ingredients 


Budgets can be an incomplete source of information about ingredients and their costs. 
Planned or projected expenditures focus on actual resources used rather than planned. 
In addition, budgets often report items not associated with a particular program, 
making it hard to attribute the correct ingredients or quantities to the program you 
are analyzing. 


However, budget information that describes expended costs can help create a list of 
ingredients and—in later steps of a cost analysis—support valuing those ingredients. 
These types of budgets can speed up cost analysis and can supplement surveys, 
interviews, or invoices. 


For a simple program, a budget with expended costs might capture all the relevant 
ingredients. For other programs, it might capture some of the ingredients and costs, 
but you will need to determine the others. 


The value of data triangulation 


Sometimes you only have time or resources to gather data about ingredients from a 
single data source. However, using multiple data sources to validate your information 
can help you be more confident about the information accuracy. For example, if 

you are administering a survey to program implementers, you might also add some 
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interviews with them, to ensure that the survey data are correct. Collecting data from 
different sites can also confirm how resources are being used. 


Describing quantity and quality 


The more descriptive you can be about a program’s ingredients, the easier it will be 
for you to value (price) those ingredients later. It will also help users of your research 
who want to replicate the program elsewhere. 


As a result, describing both the quantity and characteristics of program ingredients 

is important. To pick one example, if volunteers are used by a program, it would be 
useful to specify how many volunteers there are, how long each volunteers for, and 
what types of volunteers they are—parents? community members?—and describe what 
education levels or relevant skills they need to fulfil the role they are serving to help 
implement the program being evaluated. 


Some ingredients are used part-time by a program. Take a teacher who works halftime 
within a program, or a computer only used two hours per day by that program. Be 
sure to include in your analysis only the amount of time the relevant resource is used 
to implement your program. 


Focusing on key ingredients 


This step is about focusing your time, in terms of quantifying and describing 
ingredients, on the most important resources used by a program, not on spending a 
lot of time on small costs. A misunderstanding about cost analysis is that it requires 
researchers to meticulously account for every resource, down to the smallest detail. 


Of course, being as detailed as you can, given the time and resources available to 
devote to the cost analysis, is the goal. But small costs, such as office supplies, can 
be estimated with little impact to the cost estimate. If you are unsure whether your 
estimate for those smaller costs (or any cost) substantially affects your overall cost 
estimate, you can always run sensitivity tests to see how using different assumptions 
about quantities and prices affect the overall cost estimate. 


What are the most common key ingredients? Since most educational programs are 
labor intensive, involving teachers or other instructors, those costs will often be the 
most important. For programs that involve unsupervised computer-based learning, 
equipment costs may be the most important. 
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Categories of ingredients 


The major categories of ingredients in your analysis will depend on the program— 
there is no single, agreed upon set of categories. That said, frequently used categories 
include personnel, facilities, equipment and materials, required client inputs, and 
other inputs. That list, and the more detailed descriptions next, draw on the book 
Economic Evaluation in Education (Levin et al., 2018, pp. 62-69). 


¢ Personnel. This category includes the human resources used, including 
employees, consultants, external trainers and coaches, volunteers, and 
parents. Remember to include both time personnel spend delivering the 
program as well as their time spent in training and professional development. If 
administrator or educator time is needed to monitor the on-going success and 
improvement of the program, that time should be included as well. Personnel 
should be listed according to their roles, qualifications, and time commitments. 


¢ Facilities. Physical space, such as classrooms, offices, storage areas, training 
space or recreational facilities. It is useful to specify facilities’ dimensions and 
characteristics. 


¢ Equipment and materials. Furnishings, instructional equipment or materials 
used by the program. That might include books, manuals, computers or 
software required for implementation or training purposes. 


¢ Required client inputs. Contributions from the participants and their families. 
At the K-12 level, that could include transportation, books, uniforms or other 
items required by the program and paid for by the family. In postsecondary 
and adult education programs, it could include textbooks or other required 
resources that students need to buy. Moreover, for postsecondary and adult 
education programs, participants’ time needs to be valued, since they could 
have been using that time to earn wages (opportunity cost). 


e Other program inputs. Any other ingredients not captured by the categories 
above. That might include costs for identifying or recruiting program 
participants, transportation, prizes, food, scholarships, and financial incentives 
for participants or teachers. If the program includes oversight, monitoring, 
or tracking staff participation or student progress and outcomes, be sure to 
include the personnel costs of those efforts above and other costs here. 
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Indirect costs 


Within the categories above, there may be costs that are considered indirect, which 
are resources that are not purchased specifically for the program. Examples include 
overhead costs such as network infrastructure and internet connectivity; security, 
custodial and maintenance services; human resource functions; the work of the 
budget/finance office; or administrative costs. Where possible, these costs should be 
treated the same way in terms of quantity and price but estimating indirect costs as a 
percentage may be more feasible. 


Being aware of induced costs 


A more complicated issue is induced costs. These costs arise not as part of the main 
program model but occur because of behavioral changes caused by the program. 


For example, let us say a program uses extra quizzes to make students more aware of 
their progress. The direct costs would include the time and materials to administer 
the quizzes. But observations or interviews at the school may show that teachers using 
the method devote more time to their students, since students with low quiz scores 
come in for extra help. That extra teacher time might not have been part of the initial 
program model, but it affects program outcomes and would be considered an induced 
cost of the program. 


As you consider your program’s ingredients, look for such changes in behavior so you 
can add the resources to your ingredients. 


Continuing our working example of Summing Up 


Let us continue the example of the computer-assisted math program we introduced in 
Step 1, Summing Up. The table below shows an example of program ingredients—with 
quantity and characteristics—including program licenses, teachers’ monitoring time, 
teachers’ training time, classrooms, computers, and administrator time. 


This step should document all the program’s ingredients so that others can replicate 
the program elsewhere, regardless of whether they will reallocate ingredients or 
acquire new ones. 
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Table 6. Key Ingredients to Implement Summing Up 
Ingredient Quantity Quality/Description 


Program Single-user program licenses 
licenses for one year 


30 minutes per school day 


Teachers’ per teacher during the 
eee : Elementary school teacher 
monitoring time | academic year (180 days); 5 


teachers 
Teachers’ 4 hours per teacher; 5 


ee i Elementary school teacher 
training time teachers 


30 minutes per school day 


; 900 sq. ft elementary school 
Classrooms per classroom during the 
: classrooms 
academic year; 5 classrooms 

Dell Inspiron 21.5” Touch 
30 minutes per school Screen All-in One (AMD E2 
day per student during series with 4 GB memory 
the academic year, 100 and 1 TB hard drive) with 
computers expected lifetime use of 5 


years 


Computers 


Administrator Elementary school district 
: 5 hours as 
time administrator 


Key points 
¢ Identify and describe program ingredients to implement a program with 
fidelity 


¢ Quantify and describe ingredients in proportion to their importance to the 
program, spending most of your time on the key ingredients. 


¢ Save time and effort by building questions about ingredients into your main 
data collection strategy. 


e Make use of existing program planning and expenditure data, when available. 
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Checklist 


O I used different data collection approaches as needed to determine what 
resources were used by the program, including those that might not have been 
originally anticipated by its developer. 


O1 Ihave considered each main category of typical ingredients and identified the 
specific ingredients used by the program. 


O [have recorded program ingredients in enough detail, in terms of quantities 
and qualities, so that others will understand what it takes to implement the 
program. 


QO Iconsidered any important induced behaviors of the program and added 
relevant resources to my list of ingredients. 


Additional resources 


A sample set of interview questions is available from the CBCSE’s CostOut resource. 
Register to access the Interview Protocol under the “Additional Resources” tab. 


For more information on induced costs, see “Evaluating educational interventions 
that induce service receipt: A case study application of City Connects” in the American 
Journal of Evaluation (2016) by A. Brooks Bowden and her co-authors. 
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Phase 2: Price the ingredients 
Table 7. Building Blocks of a Cost Analysis: Phase 2 
Phase 1 Phase 2 Phase 3 


Cost 


Ingredient Quantity Description Price : ; 
(Quantity x Price) 


. Quantity in Descriptive : : 
Ingredient 1 é Quantity x Price 
units details 


: Quantity in Descriptive : : 
Ingredient 2 ’ ‘ Quantity x Price 
units details 


Overview 


While Phase 1 was about identifying the resources (ingredients) used by a program, 
Phase 2 is about placing values on, or assigning prices to, each ingredient. In Phase 3, 
you will multiply ingredient quantities by prices to obtain costs. 


What steps are involved? 


1. Identify or estimate a price for each ingredient 


2. Adjust costs for your context 
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Step 4: Identify or estimate a price for each ingredient 


Why is this important? 


In the previous step, we discussed how you identify the ingredients of the program 
you are examining. Step 4 is about assigning a price to each ingredient, which will 
allow you to calculate costs in the next phase. 


The importance of estimating both national and local prices 


Imagine you are studying a teacher professional development program in the 
Milwaukee (WI) Public Schools. In this scenario, teacher time is a key ingredient to the 
program’s success. The average hourly compensation of a teacher in Milwaukee is $X, 
while the national average is $Y. Which price should you use in your cost analysis? 


The answer? Both. 


In general, IES research informs the education field broadly, so national prices 

are most appropriate. However, because local stakeholders have real and pressing 
questions about education practice in their school or district, using local prices in 
your research makes it especially relevant. Although conducting your cost analyses 
with two parallel cost bases adds some burden to your work, it pays dividends to both 
local and national audiences. 


One word of caution: do not mix national and local prices in a single analysis. 
Accuracy and interpretation rely on consistency in your cost analysis. 


Where do I find price data? 


National prices 


If you use national prices, a fast-track method you can use is CostOut, which provides 
a database of national average prices, such as teacher and principal salaries. 
Otherwise, you can search for your own sources of national prices using, for example, 
compensation surveys produced by the Bureau of Labor Statistics. 


A second approach for generating national prices is to adjust actual local prices paid 
with a geographic price index (discussed more in step 5). You can do this in CostOut. 
This adjustment will make local prices comparable across different regions of the 
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country. If a local price is the same across the country, such as ordering a laptop 
online, no adjustment is needed. 


Local prices 


Local prices are how much an ingredient costs in a specific region. Many school 
districts publish salary schedules and salary databases for public accountability. While 
it may be your instinct to always rely upon actual salaries, be wary: any individual’s 
salary can be affected by idiosyncrasies (additional certifications, coaching 
assignments) that might cause it to deviate significantly from a local average. It may 
be better to rely upon teacher pay schedules, or average across several actual teacher 
salaries, to create your local price. 


To find facility prices, check with local realtors about comparable rents, or speak with 
builders about the cost to build similar space. Later in this section, we provide more 
information about pricing specific types of ingredients. 


Another option is to use CostOut, which allows users to adjust national prices to 
reflect geographic differences. You can adjust to a specific state or to metropolitan or 
non-metropolitan areas. The prices are still based on national averages, but they may 
better reflect the specific state or locality on which you are focusing. 


Time-saving tip: Dealing with larger quantities of ingredients 


If a program has a large quantity of any ingredient, simplifications—so long 
as you are transparent about your analysis—can be helpful. For example, if 
your program employs 20 teachers, you do not need to price each teacher 
as an individual. Instead, you could use an average price (“20 teachers at 
$65,000 each”) or disaggregate teachers into a few categories (“10 new 
teachers at $55,000 each” and “10 experienced teachers at $75,000 each”) 
rather than listing a price for each person. 


Pricing ingredients that lack market prices 


Some ingredients may not have an obvious market rate, such as classroom space or 
volunteers’ time. We share some suggestions for estimating so-called “shadow prices” 
for several common ingredients. 
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School facilities 


Some schools allow community members to rent space, which seems like a market 
price. Yet those prices might be subsidized to support the community, or higher 
than the market rate to support the school, so they are likely not very useful for cost 
analyses. 


Instead, if you are using local prices, consider these two options. One is to look at 
what a similar space would cost to rent in the private market. The other is to consider 
what it would cost to build a similarly sized space today and then amortize that 
construction cost. We will discuss amortization more in Step 5. 


If you are using national prices, the best option is to adjust a local rental rate with a 
geographic price index. If it is difficult to find a local rental rate, you can use CostOut. 


Parental or volunteer time 


Often, researchers price parents’ or volunteers’ time at the minimum wage. If 
volunteers have some specialized, relevant skill—say, they are all trained teacher 
aides— you might want to choose a higher wage. The goal is to approximate the cost of 
hiring someone to do that volunteer work. 


Student time 


The cost of K-12 students’ time is generally not included in a cost analysis, since 
these students would not be working anyway during the program. (There could be 
exceptions for high school students who could be working during program times.) 


For postsecondary and adult education programs, you generally will assign a price 
to students’ time, from both a societal and individual perspective. That is because 
those students could be working rather than being in the program. We need to count 
these costs, known as “forgone wages.” To do that, cost analyses typically assign the 
minimum wage, similar to pricing volunteer and parent time. 


Pricing other common ingredients 
Teacher time 
For teachers’ time, determine average national salaries with CostOut. If you are using 


local prices, look at the pay schedule for whichever district(s) you are working with, 
or glean information from a program budget or expenditure report. 
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Either way, for full-time teachers, you will need to decide what constitutes full-time 
work in the specific context. Ask your school district collaborators or from examining 
teacher contracts to determine expected hours of work. Alternatively, CostOut notes 
that full-time is generally defined as 2,080 hours for a typical U.S. worker, 1,440 hours 
for a K-12 teacher, and 1,260 hours for an instructor at a college or university. 


Training fees 


Training costs can involve multiple ingredients, such as teacher and trainer time, 
materials, and training space. 


It is important to note that, even though training may be planned for a program’s first 
year, its cost (and benefits) may extend longer than expected. 


First, even if all the training occurs in one year, the benefits of that training may 
last longer, with staff continuing to use the new practices they learned in training. 
Conversely, some staff may leave a multi-year program in the middle of the effort, 
which leads to training costs for their replacements. In all these cases, the costs 
should be spread over the number of years for which the training is applied by the 
staff to their practices. 


For example, if teachers get trained in year one of five, but only tend to stay in the 
school for 2.5 years, you will include new training costs for replacement staff. This 
would double the cost of training for the program and the cost of each training would 
be amortized over 2.5 years. 


Matching units 


In choosing a suitable price for each ingredient, you will need to make sure that the 
units of ingredient prices and the units of their quantities match. For example, if 
you have an hourly compensation rate for teachers, you will need to express their 
time devoted to the program in hours. You could do that by changing, say, an annual 
compensation rate to an hourly rate by dividing by the number of hours worked, 
ensuring that you express price and quantity in equivalent terms. 


A slightly trickier example is the cost of a classroom. If you know how much a 900 
square foot classroom costs to build (say, $235,000), but your units are the number 
of hours the classroom will be used in a year, you will need to spread out that cost 
over its usable life (typically 30 years). Then, once you have an annual cost for a 
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classroom ($235,000 = 30 = $7,833 using straight line amortization), you can divide 
by the number of hours in a standard school year (1,440), in a calendar year (8,760), 
or another estimate of total hours the building is available. Finally, you can take that 
hourly cost and multiply it by your hours to match your units to obtain the total cost 
of the use of the classroom. 


Continuing our working example of Summing Up 


In our example with the computer-assisted math program, Summing Up, we derived 
prices for the ingredients in Step 3. 


In Table 8, we use national prices since the goal of the analysis is to inform education 
decision makers broadly. Given our decision to use the societal perspective, we would 
determine a price for each ingredient, not just those paid for by the school or district. 


As you will see in Column 2, we also introduce the business as usual condition into 
our example to show the estimation of incremental costs. Recall that, for incremental 
costs, you only assign a price to ingredients that are used differently than in the 
business as usual or other alternative condition. 


For example, as the table shows, Summing Up uses 100 program licenses, while 
business as usual—before the program was introduced—used none. That means 

you should assign a price to program licenses and include that ingredient in your 
incremental cost analysis. Contrast that with classroom time, which is unchanged 
between the program and business as usual and therefore would not be assigned a 
price or included in the cost analysis if you are only interested in incremental costs. 


For the sake of demonstrating how to price different ingredients, we include a price 
for each ingredient in the table below, even though an incremental cost analysis 

of Summing Up would not need to price every ingredient. While Summing Up is a 
fictitious example, we discuss how we would have found the prices of its ingredients. 
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Table 8. Adding Real Prices to Summing Up 


Quantity ; 
: : Quantity : : Source/ 
Ingredient (business as ; Price per unit 
! (Summing Up) WY Crd nvexe Co) Keyea yg 
usua 


Program . : ; . 
: O licenses 100 licenses $100/license Interview 
licenses 


450 hours 450 hours $41.02/hour Costout | 


Teachers’ 
ar O hours 20 hours $41.02/hour CostOut 
training time 


450 hours 450 hours $9.65/hour /Costout 
9,000 hours $.07/hour | CostOut/ontine 


Administrator 
8 O hours 5 hours $69.69/hour CostOut 
ime 


To calculate these prices in CostOut, we set the pricing year to 2019 and took the 
following (imaginary) steps to price each resource: 


¢ Program Licenses. We found out this price by asking the Chief Technology 
Officer at the district office. 


¢ Classrooms. We were unable to find a rental rate for a similar space, which 
is the preferred method of pricing a facility. Therefore, we searched for an 
elementary school classroom in CostOut, and chose “Elementary School 
Classroom,” which has a 2015 price of $238,135. We used the recommended 
interest rate of 3.5% to amortize the cost of the classroom. For simplicity, we 
entered 1 for the quantity needed and 100% for the percentage of time the 
ingredient is used. This means that CostOut will give us the price of using a 
classroom 100 percent of the time for one year. In the column “Adj. price of 
ingredient,” which is the column we will be using, you will see $13,897. (Note, if 
you enter quantity as 1 and 100% for the percentage needed, the estimated cost 
will equal “Cost.”) The next step is to convert from a price per year to a price 
per hour. We divided $13,897 by 1440, the number of hours in a typical school 
year. We got $9.65 as an hour for an answer. 
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¢ Teachers’ monitoring time. We started with the “elementary and middle 

school teacher” price in CostOut: $1,009/week (2016 price). We used the unit 
converted in CostOut to change the salary from weekly to hourly. For the fringe 
benefit rate, we used the CBCSE database, and chose the closest description 
for an elementary school teacher. The recommended fringe rate was 53.37% of 
salary. We entered 1 as the quantity used and 100% as the percentage of time 
used. We get a cost of $41.02. We converted the units to hours and recorded the 
quantity of this ingredient in hours, so no adjustment is necessary. 


¢ Teachers’ training time. We priced the teachers’ monitoring and training time 
the same way. 


¢ Computers. For this example, we used the price of a desktop computer 
available in CostOut, labelled “desktop computer.” It has a 2018 price of 
$449.99. Because the price of a computer does not vary by region, it is also 
an option to look up the price of a computer online. We assumed that the 
computer would last five years and used the recommended discount rate of 
3.5%. We entered 1 as the quantity used and 100% as the percentage of time 
the ingredient is used. CostOut gives us a price of $100.98. But this is the 
yearly price of a computer, and we measured computer use in hours. Much 
like teacher time, we divide $100.98 by 1440 hours, the number of hours in a 
typical school year. We got 7 cents as an hourly rate. 


e Administrator time. We started with the price (converted to hours) of an 
“administrative services managers - elementary and secondary schools” 
in CostOut. It is a 2015 price of $42.40 an hour. For the fringe benefit rate, 
we relied on the CBCSE database, and chose the closest description for an 
elementary school teacher. The recommended fringe rate was 53.37% of salary. 
We entered 1 as the quantity used and 100% as the percentage of time the 
ingredient is used. 


Note that in CostOut, you will see that prices listed in the database are from several 
different years. Setting the pricing year in CostOut will cause it to inflation-adjust any 
of the prices you select regardless of its year of origin. 


Incremental costs vs. total costs of implementation 


In Summing Up, “business as usual” was a standard math class and we showed costs 
of our program that were incremental to that class. Some programs, especially 
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after-school and summer programs, will not be replacing any alternative. For those 
programs, business as usual is no program at all, so you would price all ingredients, 
producing total implementation costs rather than incremental costs. 


If a program takes place across multiple sites, business as usual might vary. If that is 
the case, you would need to compare the program to its multiple alternatives. If you 
think that the alternatives are similar, or if you have limited resources to examine 
every site, you might choose a sample of sites as a comparison. If you know that sites 
are likely to have different business as usual conditions, it is best to visit each site to 
properly identify and price the resources in use. 


Key points 


e Although cost analyses based in national prices have a different purpose 
from those based in local prices, both can be important to consumers of your 
research. National prices are more likely to generalize, while local prices are 
more likely to be relevant to your immediate stakeholders. 


e Even if you choose local prices, using market prices (when different from actual 
prices) will help reduce unwanted “noise” from your cost estimate. 


¢ Which ingredients of a program you will need to assign prices to will depend 
on the perspective(s) you choose for the cost analysis. Using the recommended 
societal perspective means you will need to place values on all ingredients, 
assigning a price for each one. Narrower perspectives require less pricing but 
may yield less useful data in the long run. 


¢ CostOut provides national prices on a wide range of ingredients. It also allows 
you to adjust national prices to a local level, if you want to use local prices. 


Checklist 


O Iam prepared to conduct parallel cost analyses using both national and local 
price data. 


O1 If there is a constraint on my choice of prices, I have determined whether 
national or local prices make the most sense given the specific purpose and 
target audience of my cost analysis. 
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O [have assigned prices to each ingredient in my analysis using CostOut or other 
sources. 


O [have ensured that units of both price and quantity align. 


Additional resources 


A useful online overview of pricing ingredients is provided by Robert Shand, as part of 
an IES-sponsored workshop by CBCSE, and is available here. 
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Step 5: Adjust costs for your context 


Why is this important? 


By this point, you have identified the ingredients and have assigned prices to them. 
You now have enough information for a basic cost estimate: quantity ~ price. 


In Step 5, we focus on adjusting these costs to improve their accuracy. These include 
adjusting for the geographic location, amortizing capital costs, and discounting costs 
in the later years of a multi-year program. You may already have adjusted some costs 
in Step 4 but pay attention to the common adjustments below. 


How to make the adjustments 


CostOut prompts you to enter the relevant information needed for all adjustments 
discussed in this section and does the math for you. Using CostOut ensures you will 
not mistakenly overlook an adjustment, since it will prompt you for each one. 


Another possibility is to calculate the adjustments yourself. You can consult the short 
descriptions below of the key adjustments, as an overview. For more information, 
including any formulas involved, you will want to consult other sources. One useful 
source is the manual provided within CostOut (available after free registration), even if 
you do not use the software to make the calculations. 


The adjustments include— 


e Adjustments for geographic differences. This is useful because the cost of 
ingredients will vary between locations—the cost of space, for instance, will 
likely be different in an urban versus a rural area. One reason to adjust for 
geographic differences is if you are unable to find local prices, you may want to 
adjust from a national price available on CostOut. The other reason is to make 
local prices more comparable. If you were comparing the cost of a program in 
an urban and rural area, you could adjust both prices to eliminate geography- 
related factors. 


¢ Adjustments for multiyear capital costs. This process is known as 
amortization and refers to the process of spreading out the cost over an asset’s 
useful life. For instance, if you buy a computer in year one of a three-year 
program, you will not want to count the full cost of that computer in just the 
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first year. Rather, you would estimate the useful life of a computer (say, five 
years) and divide the total cost by five. Then, you would include three of those 
years as the cost of the computer for the program, but only a third of this in 
year one. For more complicated amortizations that account for the interest 
forgone on the dollar amount invested in a capital item—especially facilities—we 
recommend using the CostOut tool. 


e« Adjustments for ingredients used for multiple purposes. Some ingredients 
are used part-time by a program. Take a teacher who works halftime within a 
program, or a computer only used two hours per day by that program. This 
step is simply about multiplying a resource’s cost by the fraction of time it is 
used by the program. You may have already done this for some ingredients, 
such as if you already calculated the number of hours a teacher is involved in a 
program rather than listing their total annual hours. 


¢ Adjustments to reflect present value. This adjustment is important when 
you are estimating costs for programs that last multiple years. Dollar amounts 
in later years are valued less than amounts in the base year due to the time 
value of money—for example, a dollar next year is worth less than a dollar 
today because you cannot invest it now and earn interest on it. (The same is 
true for benefits.) To take that time value of money into account, we adjust 
costs incurred after the base year by what is called a discount rate. Choosing a 
discount rate is one step in this process. CostOut recommends a discount rate of 
3.5%. (It is acceptable for IES grantees to use this discount rate.) The process of 
doing that adjustment is called a present value calculation. Programs that only 
last a year can skip this adjustment. 


Key points 


¢ Once you have identified a program’s ingredients and assigned prices to them, 
a series of adjustments to costs can improve their accuracy for your context. 


e You can use CostOut to make the adjustments or do them on your own. 


Checklist 


O Ihave made contextual adjustments, such as geographic or present value 
adjustments, relevant to the program I am examining. 
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Phase 3: Create & use the cost estimate 


Table 9. Building Blocks of a Cost Analysis: Phase 3 


Cost 
Ingredient uantit Description 
er Q y P (Quantity x Price) 


: Quantity in Descriptive ; . 
Ingredient 1 : : Quantity x Price 
units details 


; Quantity in Descriptive : : 
Ingredient 2 ; ‘ Quantity x Price 
units details 


Overview 


The final phase of a cost analysis is about putting together quantity and price 
information, including any adjustments, to create a cost estimate. It also involves 
conducting any necessary sensitivity analyses to test important assumptions in 
your calculations to see how they affect that estimate. Finally, you can use the cost 
estimate, as needed, as part of more advanced forms of economic evaluation. 


What steps are involved? 


1. Calculate costs 
2. Conduct sensitivity analysis 


3. Determine whether to use the results for further economic evaluation 
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Step 6: Calculate costs 


Now that we have discussed how to identify and price ingredients, it is time to put 
everything together. The most common approach is to calculate a total cost and a cost 
per participant. 


Why is this important? 


This step is about creating your cost estimate. It is also about thinking about what 
decision makers need to know and ensuring that what you present fits their needs. 


How to present your cost estimate 


To calculate the cost of each ingredient, you will multiply the quantities you identified 
in Step 3 by the per-unit price you found in Step 4 or the adjusted price from Step 

5. As part of Step 4, you matched the units of your ingredients to the units of your 
prices. Make sure to use those same matching units as you present total costs. You will 
need the costs of each ingredient to create the different summary cost metrics below. 
We have included a simplified example using the hypothetical Summing Up math 
program, to explain how each metric is calculated. 


Table 10 shows how to calculate the total program costs for business as usual and 
Summing Up, as well as the incremental costs for Summing Up. For Total Cost, you add 
together the costs of every ingredient required for the business as usual condition and 
for Summing Up. For the Incremental Cost of Summing Up, you only add together the 
ingredients beyond what is needed for the business as usual (as described in Table 

8). These incremental costs may be a whole new ingredient (for example, program 
licenses) or it may be the same ingredient but with a different amount used (for 
example, hours of computer time). In addition to determining the incremental cost of 
the intervention, adding up the costs in this way also allows you to— 


e Understand how much you are spending on business as usual 


e Determine the cost feasibility of the intervention based on both its total and 
incremental cost 


e Determine the incremental cost of the intervention in two ways: 


» identify the costs of ingredients not used in the business as usual but used in 
the intervention 
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» identify the total costs of the business as usual and the total costs of the 
intervention and then subtract them (this is useful when there may be 
differences of opinion on which ingredients or which portions of them are 
incremental costs of the intervention). 


Table 10. Calculating Total Costs and Incremental Costs for Summing Up 


Phase 1 Phase 2 


Price ; . Incremental 
Ingredient Quantity Total cost Quantity Total Cost 
(per unit) Cost 


Program 
; $100.00/ea. $0.00 100 $10,000.00 | $10,000.00 
licenses 
Teacher 
classroom $41.02/hr. 450 $18,459.00 450 $18,459.00 $0.00 
time 


Teacher 
ae $41.02/hr. $0.00 $820.40 $820.40 
training time 


Administrator 
: $69.49/hr. $0.00 $347.45 $347.45 
ime 


* Calculated by subtracting the total cost of business as usual from the total cost of Summing 
Up. 


Costs per participant 


Once you have your total program costs, you can divide that estimate by the number 
of participants served by your program. This will give you a cost per participant, 
which is a useful metric when comparing costs between programs. In our example, 
100 students received the program so the calculation would be— 
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Total costs + number of participants = cost per participant 


$11,796.10 + 100 = $117.96 

Costs by payer 

If you took a societal perspective, which includes the costs of every ingredient of the 
program, one useful option is to create a table that assigns each cost to a stakeholder 
that pays for that ingredient—you might call them the “payer.” (Note that some of 


those costs are opportunity costs.) Doing this gives decision makers a clear picture of 
who bears a share of the cost of an intervention. 


Table 11. Summing Up’s cost totaled by payer 


; ; Price (per 
Ingredient Quantity Payer ; Cost 
unit) 


Program . oe 
; 100 licenses District $100.00 $10000.00 
Licenses 
Teachers’ 

Pe ene O hours* School 
monitoring time 
Teachers’ 

20 hours School $41.02 $820.40 

=a time 


Classrooms ee Ohours* School 


Administrator’s 
§ 5 hours District $69.49 $347.45 
ime 


* These quantities are set to zero because they are the same as business as usual. 


** 9,000 hours for Summing Up minus 25 hours for business as usual 
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Costs by year 


For programs that last more than one year, you need to determine whether ingredient 
use, and the costs of those ingredients, will change from year to year. If they do not, 
you can estimate costs for one year only and report that these will be repeated in later 
years. If the costs vary across years, you should present costs for each year in addition 
to a total across all years. 


Key points 


¢ Create your cost estimate by bringing together the ingredients and quantities 
you identified in Phase 1 with the prices and adjustments you found in Phase 2. 


e The most common approaches to presenting costs are total program cost and 
cost per participant. 


e Your target audience will help determine how you prepare and present your 
cost estimate, but we recommend showing costs by ingredient category, by 
payer, and by year for multiyear programs. 


Checklist 


O Ihave presented costs in a way that fits the needs of my audience. 


O If have a multiyear program, I have considered whether the costs change year 
to year and presented the results accordingly. 


Additional resources 


Along with this starter toolkit, there is also a starter spreadsheet that will help you 
walk through your ingredients and record your total program costs. You can search 
for both it and this publication by number via the IES Publication Search feature. 


For more information on how costs can be aggregated and reported to decision 
makers, see “Guiding the Development and Use of Cost-Effectiveness Analysis in 
Education” in the Journal of Research on Educational Effectiveness (2015) by Henry 
Levin and Clive Belfield. 
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Step 7: Conduct sensitivity analysis 


Why is this important? 


As you build your cost estimate in the steps above, you will have made certain 
assumptions about the quantity of your ingredients or their characteristics, the 
number of participants served, the number of years over which to spread costs and 
the related interest rate, and the discount rate for multiyear programs. Sensitivity 
analysis is about varying those assumptions to see if different choices would result in 
significant differences to your cost estimate. While not a requirement of conducting a 
cost analysis, sensitivity analysis can give more credibility to your estimates and is a 
best practice. 


What kinds of sensitivity analysis are most useful? 


The most useful types of sensitivity analysis when analyzing costs of educational 
programs involves testing out different values for prices or quantities of the key 
ingredients in your program and different numbers of participants being served. 


Teacher time is the most common (and often the most expensive) ingredient in 
educational programs, so testing different assumptions about the amount and unit 
costs of labor may be particularly valuable. Those assumptions might involve salaries, 
teacher qualifications, or the number of teachers or hours of teacher time. 


Other ingredients may call for exploration based on the nature of your program. For 
example, a technology-based program may have high capital costs, so you may want 
to test any assumptions related to the computers or other technology involved. 


One bit of advice: Do not get hung up if you are not sure about the use or price 

of a specific ingredient. Even experienced cost analysts sometimes disagree on a 
standard approach, such as whether to price volunteers or whether to assign costs to 
a classroom that is not otherwise in use. The best advice is to pick an approach, be 
transparent in your write-up, and use sensitivity analysis to determine if any of your 
choices make a substantial difference to the program’s cost estimate. If they do, report 
that in your write-up. 
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An example 


You may know that district administration will spend time overseeing your program, 
but not know how much of their time they will dedicate or exactly whose time it 

will be. To find a reasonable estimate for district staff monitoring time, you could 
develop a range of hours or salaries that seem reasonable or are based on similar 
programs. Doing that might result in high, medium, and low estimates of hours/salary. 
Your next step would be to calculate costs of administrator time using each of these 
three estimates to determine if choosing one or another of those options will result 

in important differences to your overall costs. If so, you will want to be transparent 
about that in your write-up. 


Key points 
¢ Sensitivity analysis helps you understand to what extent assumptions you have 
made impact the results of your cost analysis. 


e Your sensitivity analysis should focus on key ingredients about which there is 
the greatest uncertainty regarding assumptions about quantity and price. 


e Labor is often the biggest cost driver for educational programs, but this may 
not hold true for every program. 


Checklist 


O1 Ihave considered the biggest cost drivers of my program and whether they 
should be included in a sensitivity analysis. 
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Step 8: Determine whether to use the results for 
further economic evaluation 


Depending on the questions you are trying to answer with your cost analysis, more 
steps to assess cost-feasibility or return on investment may be necessary. 


If you are only interested in documenting the resources required to implement 
your program and their associated costs, Steps 1 - 7 in this guide will give you the 
information you need. 


However, if you have other cost questions—or if IES requires you to do so—you will 
need to take further steps. Be sure to check the specific requirements of your grant 

by referencing the current Requests for Applications (which can be accessed from the 
IES funding page). We recommend reviewing the requirements and recommendations 


for your grant before taking Step 8, so you can pay closer attention to the type of 
economic evaluation that applies to your research. 


Why is this important? 


When a cost analysis is completed, it can provide useful information by itself about 
program costs. For example, some research questions that could be examined with 
cost analysis include— 


e What resources are needed to replicate the ABC early intervention in math? 


e What is the cost of the XYZ supplemental reading program? 


e¢ What part of the cost of the EFG college counseling intervention is borne by 
students? 


However, cost analysis often serves as a building block for other forms of economic 
analysis. You can think of these additional economic evaluations as cost analysis 
“plus”, where you apply all of the knowledge you have gained in Steps 1 - 7, and take 
additional steps. We explain major approaches below. It is important to underscore 
that each of them starts with a high-quality cost analysis. 


Common types of economic evaluation 


The three most common types of economic analysis in education research are 
described below. When selecting a method, keep in mind your grant’s requirements 
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(listed in the Request for Applications). Also consider the advantages and 
disadvantages of each, described later in this section. 


Cost-feasibility analysis (CFA) 


This relatively simple approach takes the results of a cost analysis for a program and 
compares it to the resources available within, say, a school or district. 


Cost-feasibility analysis is used to answer the question, “Can we afford this program?” 
For example, if a cost analysis shows that a certain program costs $100 per high school 
student, the principal could use that information to consider whether the school can 
afford to adopt the program. 


Cost-effectiveness analysis (CEA) 


This approach runs cost analyses on alternative programs, estimating the incremental 
costs of each program compared to the business as usual condition. It then divides 
each program’s incremental costs by the effect size that each produces when 
compared to the common baseline condition, assuming that effects are measured the 
same way across programs. 


This type of analysis answers questions such as, “Which program achieves a given 
level of improvement at the lowest cost?” or “Which program achieves the most 
improvement for a given cost?” For example, Program A costs $4.30 per student and 
produces an average gain above and beyond the baseline condition of 3.55 points on 
a national test. That is a cost-effectiveness ratio of $1.21 (or $1.21 for each point gain 
above and beyond the baseline condition on the national test). Program B, on the 
other hand, costs $20.85 per student and produces an average gain above and beyond 
the baseline condition on the same test of 7.23 points. That is a cost-effectiveness ratio 
of $2.88 (or $2.88 for each point gain above and beyond the baseline condition on the 
national test). These results show that Program A is more than twice as cost-effective, 
given that a lower ratio means less cost per unit of gain. You get an average of a .82 
point per dollar spent on Program A, but only a .34-point gain for each dollar invested 
in Program B. 


Cost-benefit analysis (CBA) 
This approach, also called benefit-cost analysis, compares a program’s societal costs to 


a monetized value of its benefits to assess whether the benefits exceed the costs. You 
can also do this for multiple programs, regardless of the outcome measure. 
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This type of analysis helps answer the question, “Should we invest in this program—or 


which among several programs should we invest in?” For example, the incremental 


costs of a dropout program compared to business as usual at Site A are $70,560 and 


it produces benefits (the number of graduates above and beyond business as usual 
times the financial benefits to society of an averted dropout) of $344,000. That is a 


benefit/cost ratio of $4.90. At Site B, the incremental program costs are $214,200 and 
the program produces benefits of $309,600. That is a benefit/cost ratio of $1.40. These 
results show that Program A produces approximately 3.5 times the economic benefits 


as Program B. 


Table 12. Comparison of economic evaluation methods 


Formula 


Interpretation of 
Results 
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Cost-feasibility 


Budget amount 
available minus 


costs of program 


If > O, you can 
afford the program 
If < O, you do not 
have the resources 
for the program 


Useful to 


districts Easy to 


understand/relay to 
stakeholder 
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Cost-effectiveness 


Incremental costs + 


effect size 


Higher numbers are 
less cost-effective 
when comparing 
CEA ratios 


Useful to school 
decision makers 
and policy makers 
Comparable across 
programs with the 
same effectiveness 


measure 


Cost-benefit 


Monetized benefits 
minus incremental 
costs, or 

Monetized benefits 


+ incremental costs 


If benefits - costs > 
O, or if benefits + 
costs > 1, it means a 
positive return on 


investment 


Useful to policy 
makers 
Comparable 
across programs, 
regardless of 
effectiveness 


measure 
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Cost-feasibility Cost-effectiveness Cost-benefit 


Programs must Societal benefits 
Does not include have common can be difficult to 
Cons any type of ROI outcomes for price especially for 
analysis CEA ratios to be programs serving 
comparable young students 


Key points 


e¢ The economic evaluation method you choose should be based on your research 
needs, the needs of your audience, and any applicable IES requirements. 


Checklist 


O Iam aware of both strengths and weaknesses of the economic evaluation 
methods I am using. 


O [have referenced the current IES Requests for Applications and I know if there 
are requirements for a specific type of economic evaluation for my grant. 


Additional Resources 


Step 8 provided only a brief overview of a few additional forms of economic 
evaluation. There are a number of resources available for those who need to expand 
on their cost analysis. The Regional Education Labs provide a useful overview of 
the economic evaluation methods covered here, with additional examples. We also 
recommended referencing CBCSE’s training videos, particularly, Clive Belfield’s 
overview of CFA, CEA, and BCA. 


While cost-feasibility is a straightforward method of comparing your cost analysis 

to your budget, CEA and BCA are more technical. The Abdul Latif Jameel Poverty 
Action Lab (J-PAL) provides useful information for those conducting cost-effectiveness 
analysis. For BCA, the Harvard T.H. Chan School of Public Health’s Reference Case 
Guidelines for Benefit-Cost Analysis is a useful resource. While Harvard’s guide is 
targeted to health research, many of the same concepts can be applied to BCA for 
education research. 
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Appendix A: Logic Model Template 


AE Outcomes 
Inputs Activities Outputs : 
Short term Medium term Long term 
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Appendix B: Master List of Additional Resources 


From Introduction 


The set of free online videos of an IES-funded workshop on cost analysis organized 
by the Center for Benefit-Cost Studies of Education (CBCSE) at Teachers College, 
Columbia University. They include— 


e The Ingredients Method Part 1 and Part 2 by A. Brooks Bowden 
¢ Pricing Ingredients by Robert Shand 


From STEP 1: Clarify the program model 


e¢ A blank logic model template is provided in Appendix A that you can print and 
use. 


e Asummary of additional resources on logic models created by the IES Regional 
Education Laboratories (RELs) is provided here 


¢ To see other formats for logic models that you might want to consider, such as a 
flow chart, visit this REL resource provided here. 


From STEP 3: Identify the quantity and quality of the program’s key 
ingredients 


e Asample set of interview questions is available from the CBCSE’s CostOut 
resource. Free registration is required to access the Interview Protocol. 
Once you log in, you can find the interview protocol under the “Additional 
Resources” tab. 


¢ For more information on induced costs, see “Evaluating educational 
interventions that induce service receipt: A case study application of City 
Connects” in the American Journal of Evaluation (2016) by A. Brooks Bowden 
and her co-authors. 


From STEP 4: Identify or estimate a price for each ingredient 


¢ A useful online overview of pricing ingredients is provided by Robert Shand, as 
part of an IES-sponsored workshop by CBCSE, and is available here. 
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From STEP 6: Calculate costs 


Along with this starter toolkit, there is also a starter spreadsheet that will help 
you walk through your ingredients and record your total program costs. 


For more information on how costs can be aggregated and reported to decision 
makers, see “Guiding the Development and Use of Cost-Effectiveness Analysis 
in Education” in the Journal of Research on Educational Effectiveness (2015) by 


Henry Levin and Clive Belfield. 


From STEP 8: Determine whether to use the results for further economic 
evaluation 


Step 8 provided only a brief overview of a few additional forms of economic 
evaluation. There are a number of resources available for those who need to 
expand on their cost analysis. The Regional Education Labs provide a useful 
overview of the economic evaluation methods covered here, with additional 
examples. We also recommended referencing CBCSE’s training videos, 
particularly, Clive Belfield’s overview of CFA, CEA, and BCA. 


While cost-feasibility is a straightforward method of comparing your cost 
analysis to your budget, CEA and BCA are more technical. The Abdul Latif 
Jameel Poverty Action Lab (J-PAL) provides useful information for those 
conducting cost-effectiveness analysis. For BCA, the Harvard T.H. Chan School 
of Public Health’s Reference Case Guidelines for Benefit-Cost Analysis is a useful 
resource. While Harvard’s guide is targeted to health research, many of the 
same concepts can be applied to BCA for education research. 
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Appendix C: Credits 


The content in this guide, originally created by Grant Thornton Public Sector LLC, 
integrates concepts and examples drawn heavily from several important sources of 
information on economic evaluation. These include— 


¢ The book, Economic Evaluation in Education (third edition) by Henry Levin, 
Patrick McEwan, Clive Belfield, A. Brooks Bowden and Robert Shand, published 
by Sage in 2018. 


¢ The CostOut online tool created by the Center for Benefit-Cost Studies of 
Education (CBCSE) at Teachers College, Columbia University, funded by IES 
and accessible at no cost. 


e Videos of a workshop on economic evaluation provided by CBCSE in 2017 that 
are available online, especially the following modules: 


» Pricing Ingredients by Robert Shand 
» The Ingredients Method, Parts 1 and 2, by A. Brooks Bowden. 


¢ Resources on the “Conducting Cost-Effectiveness Analysis” webpage from the 
Abdul Latif Jameel Poverty Action Lab (J-PAL). 


The starter kit was also informed by interviews conducted for this project with leaders 
in the costing field related to education and more broadly. Particularly central to this 
work was input from Rebecca Maynard, A. Brooks Bowden of the Graduate School of 
Education at the University of Pennsylvania, Robert Shand of the School of Education 
at American University, and Fiona Hollands of Teachers College, Columbia University. 
Other interviewees provided valuable input and they are listed in Appendix D. 


We are grateful for their generosity with their time and insights. 


IES RV Institute of = 
my Education Sciences 


Appendix D: Interviewees in Developing This Starter 
Toolkit 


A. Brooks Bowden, Graduate School of Education, University of Pennsylvania 
Al McGartland, Environmental Protection Agency 

Alexandra Resch, Mathematica 

Anthony Pember, Grant Thornton Public Sector 

Anu Dathan, Abdul Latif Jameel Poverty Action Lab (J-PAL) 

Ashok Vadgama, Consortium for Advanced Management International (CAM-I) 


Bill Dummer, Consortium for Advanced Management International (CAM-I) and 
Boeing 


Burt Barnow, Trachtenberg School of Public Policy, George Washington University 
Caitlin Tulloch, International Rescue Committee 

Catherine Bradshaw, University of Virginia 

Clive Belfield, Queens College, City University of New York 

Derek Sandison, CAM-I and Sapling Corporation 

Fiona Hollands, Teachers College, Columbia University 

Henry Levin, CBCSE, Teachers College, Columbia University 

Jaunelle Pratt-Williams, SRI International 

Jon Kay, Education Endowment Foundation, United Kingdom 

Kerry Krutilla, Indiana University, O’Neill School of Public and Environmental Affairs 
Lynn Karoly, Pardee RAND Graduate School 


Max Crowley, Pennsylvania State University, College of Health and Human 
Development 


Radhika Bhula, Abdul Latif Jameel Poverty Action Lab (J-PAL) 
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Rebecca Maynard, Graduate School of Education, University of Pennsylvania 
Robert Shand, American University, School of Education 
Shiva Verma, Grant Thornton Public Sector 


Stephanie Riegg Cellini, Trachtenberg School of Public Policy, George Washington 
University 
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